{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 用户数据处理\n",
    "(只取训练集和测试集中出现的用户 ID)\n",
    "\n",
    "数据来源于 Kaggle 竞赛: Event Recommendation Engine Challenge, 根据\n",
    "①events they’ve responded to in the past\n",
    "②user demographic information\n",
    "③what events they’ve seen and clicked on in our app\n",
    "预测用户对某个活动是否感兴趣\n",
    "\n",
    "竞赛官网:\n",
    "https://www.kaggle.com/c/event-recommendation-engine-challenge/data\n",
    "\n",
    "用户描述信息在 users.csv 文件: 共 7 维特征  \n",
    "user_id: 用户 ID  \n",
    "locale: 地区, 语言  \n",
    "birthyear: 出生年份  \n",
    "gender: 性别  \n",
    "joinedAt: 用户加入 APP 的时间, ISO-8601 UTC 时间  \n",
    "location: 地点  \n",
    "timezone: 时区"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy.sparse as ss\n",
    "import scipy.io as sio\n",
    "# 保存数据\n",
    "import pickle\n",
    "# user 的特征需要编码\n",
    "from utils import FeatureEng\n",
    "# 归一化\n",
    "from sklearn.preprocessing import normalize\n",
    "# 相似度/距离\n",
    "import scipy.spatial.distance as ssd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总的用户数目超过训练集和测试集中的用户, \n",
    "为节省处理时间和内存, 先去处理 train 和 test, 得到竞赛需要用到的活动和用户. \n",
    "然后对在训练集和测试集中出现过的活动和用户建立新的 ID 索引. \n",
    "先运行 user_event.ipynb,\n",
    "得到用户列表文件: PE_userIndex.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取之前统计好的测试集和训练集中出现过的用户"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users in train & test: 3391\n"
     ]
    }
   ],
   "source": [
    "# 读取训练集和测试集中出现过的用户列表\n",
    "userIndex = pickle.load(open('PE_userIndex.pkl', 'rb'))\n",
    "n_users = len(userIndex)\n",
    "print('Number of users in train & test: %d' %n_users)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>locale</th>\n",
       "      <th>birthyear</th>\n",
       "      <th>gender</th>\n",
       "      <th>joinedAt</th>\n",
       "      <th>location</th>\n",
       "      <th>timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3197468391</td>\n",
       "      <td>id_ID</td>\n",
       "      <td>1993</td>\n",
       "      <td>male</td>\n",
       "      <td>2012-10-02T06:40:55.524Z</td>\n",
       "      <td>Medan  Indonesia</td>\n",
       "      <td>480.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3537982273</td>\n",
       "      <td>id_ID</td>\n",
       "      <td>1992</td>\n",
       "      <td>male</td>\n",
       "      <td>2012-09-29T18:03:12.111Z</td>\n",
       "      <td>Medan  Indonesia</td>\n",
       "      <td>420.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>823183725</td>\n",
       "      <td>en_US</td>\n",
       "      <td>1975</td>\n",
       "      <td>male</td>\n",
       "      <td>2012-10-06T03:14:07.149Z</td>\n",
       "      <td>Stratford  Ontario</td>\n",
       "      <td>-240.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1872223848</td>\n",
       "      <td>en_US</td>\n",
       "      <td>1991</td>\n",
       "      <td>female</td>\n",
       "      <td>2012-11-04T08:59:43.783Z</td>\n",
       "      <td>Tehran  Iran</td>\n",
       "      <td>210.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3429017717</td>\n",
       "      <td>id_ID</td>\n",
       "      <td>1995</td>\n",
       "      <td>female</td>\n",
       "      <td>2012-09-10T16:06:53.132Z</td>\n",
       "      <td>NaN</td>\n",
       "      <td>420.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id locale birthyear  gender                  joinedAt  \\\n",
       "0  3197468391  id_ID      1993    male  2012-10-02T06:40:55.524Z   \n",
       "1  3537982273  id_ID      1992    male  2012-09-29T18:03:12.111Z   \n",
       "2   823183725  en_US      1975    male  2012-10-06T03:14:07.149Z   \n",
       "3  1872223848  en_US      1991  female  2012-11-04T08:59:43.783Z   \n",
       "4  3429017717  id_ID      1995  female  2012-09-10T16:06:53.132Z   \n",
       "\n",
       "             location  timezone  \n",
       "0    Medan  Indonesia     480.0  \n",
       "1    Medan  Indonesia     420.0  \n",
       "2  Stratford  Ontario    -240.0  \n",
       "3        Tehran  Iran     210.0  \n",
       "4                 NaN     420.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "users = pd.read_csv('users.csv')\n",
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 38209 entries, 0 to 38208\n",
      "Data columns (total 7 columns):\n",
      "user_id      38209 non-null int64\n",
      "locale       38209 non-null object\n",
      "birthyear    38209 non-null object\n",
      "gender       38100 non-null object\n",
      "joinedAt     38152 non-null object\n",
      "location     32745 non-null object\n",
      "timezone     37773 non-null float64\n",
      "dtypes: float64(1), int64(1), object(5)\n",
      "memory usage: 2.0+ MB\n"
     ]
    }
   ],
   "source": [
    "users.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "gender, joinedAt, location, timezone 有缺失值."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 处理 users.csv --> 特征编码、用户之间的相似度\n",
    "FE = FeatureEng()\n",
    "n_cols = users.shape[1] -1 # 去掉 user_id 列\n",
    "cols = ['LocaleId', 'BirthYearInt', 'GenderId', 'JoinedYearMonth', 'CountryId', 'TimezoneInt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# users 编码后的特征\n",
    "userMatrix = ss.dok_matrix((n_users, n_cols))\n",
    "for u in range(users.shape[0]):\n",
    "    userId = str(users.loc[u, 'user_id']).encode(encoding='utf-8')\n",
    "    \n",
    "    if userIndex.__contains__(userId):\n",
    "        i = userIndex[userId]\n",
    "        \n",
    "        userMatrix[i, 0] = FE.getLocaleId(users.loc[u, 'locale']) # 地区, 语言\n",
    "        userMatrix[i, 1] = FE.getBirthYearInt(users.loc[u, 'birthyear']) # 出生年份\n",
    "        userMatrix[i, 2] = FE.getGenderId(users.loc[u, 'gender']) # 性别\n",
    "        userMatrix[i, 3] = FE.getJoinedYearMonth(users.loc[u, 'joinedAt']) # 注册月数\n",
    "        userMatrix[i, 4] = FE.getCountryId(users.loc[u, 'location']) # 国家 由于写法不规范, 可能该编码不起作用\n",
    "        userMatrix[i, 5] = FE.getTimezoneInt(users.loc[u, 'timezone']) # 时区"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 归一化用户特征矩阵\n",
    "userMatrix = normalize(userMatrix, norm='l2', axis=0, copy=False)\n",
    "sio.mmwrite('US_userMatrix', userMatrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 计算用户相似度矩阵, 用于之后用户推荐系统\n",
    "userSimMatrix = ss.dok_matrix((n_users, n_users))\n",
    "# 读取在训练集和测试集中出现的用户对\n",
    "uniqueUserPairs = pickle.load(open('FE_uniqueUserPairs.pkl', 'rb'))\n",
    "# 对角线元素\n",
    "for i in range(0, n_users):\n",
    "    userSimMatrix[i, i] = 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 对称\n",
    "for u1, u2 in uniqueUserPairs:\n",
    "    i = u1\n",
    "    j = u2\n",
    "    if not userSimMatrix.__contains__((i, j)):\n",
    "        usim = ssd.correlation(userMatrix.getrow(i).todense(), \n",
    "                              userMatrix.getrow(j).todense())\n",
    "        userSimMatrix[i, j] = usim\n",
    "        userSimMatrix[j, i] = usim\n",
    "        \n",
    "sio.mmwrite('US_userSimMatrix', userSimMatrix)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
